Skip to content

fix: stop pinning live DataLoaders in quantized model state - #20

Merged
Adithya-Thonse merged 4 commits into
TexasInstruments:mainfrom
musicalplatypus:pr/dataloader-pinning-fix
Aug 6, 2026
Merged

fix: stop pinning live DataLoaders in quantized model state#20
Adithya-Thonse merged 4 commits into
TexasInstruments:mainfrom
musicalplatypus:pr/dataloader-pinning-fix

Conversation

@musicalplatypus

Copy link
Copy Markdown
Contributor

Summary

TinyMLQuantFxBaseModule stored the full qconfig_type dict — including the raw calibration/eval DataLoader objects auto-quantization embeds for its one-time bitwidth search — as self.qconfig_type, permanently reachable from the wrapped model.

Root cause

With persistent_workers=True and workers already spun up, copy.deepcopy(model) in export_model() hits a live _MultiProcessingDataLoaderIter and crashes:

NotImplementedError: ('{} cannot be pickled', '_MultiProcessingDataLoaderIter')

A second, related bug found while fixing the first: self.qconfig_type = qconfig_type is a reference assignment, so the wrapper module's dict and the caller's dict were the same object. The subsequent .pop('calibration_dataloader') / .pop('eval_dataloader') mutated whatever dict the caller passed in, not just the wrapper's own copy — a caller that constructs qconfig_type once and reuses or reads it afterward would see it silently emptied by a constructor call it doesn't own.

Fix

  • Drop the DataLoader references from self.qconfig_type once the auto-quantization search has consumed them, since nothing reads them afterward.
  • Copy qconfig_type before popping from it, so the caller's own dict is never mutated.
  • Wire tinyml-modeloptimization/torchmodelopt into CI: the regression test for this was never actually executed — the workflow only triggered on tinyml-modelmaker/** paths and had no step to run torchmodelopt's own tests, so a regression here could merge without a red check.

Verification

Added test_quant_base_dataloader_leak.py, covering both the deepcopy-after-export crash and the caller-dict-mutation case.

Note: the CI workflow change here touches .github/workflows/test-modelmaker.yml, which two of my other open PRs (pr/test-ci-hygiene, pr/mps-eval-fixes) also touch — each adds a path-trigger/step at the same position. If merged after either, GitHub will show a trivial conflict; the correct resolution is keeping all the added lines (a union), not choosing one over the other.

🤖 Generated with Claude Code

t5fkg8d44d-beep and others added 3 commits July 29, 2026 11:15
…rt deepcopy

TinyMLQuantFxBaseModule stored the full qconfig_type dict (including the
raw calibration/eval DataLoader objects auto-quantization embeds for its
one-time bitwidth search) as self.qconfig_type, permanently reachable from
the wrapped model. With persistent_workers=True and workers already spun
up, copy.deepcopy(model) in export_model() would then hit a live
_MultiProcessingDataLoaderIter and crash with
NotImplementedError: ('{} cannot be pickled', '_MultiProcessingDataLoaderIter').

Drop the dataloader references from self.qconfig_type once the
auto-quantization search has consumed them, since nothing reads them
afterward.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… CI on its changes

The regression test added for the DataLoader-pinning fix
(tinyml-modeloptimization/torchmodelopt/tests/test_quant_base_dataloader_leak.py)
was never executed by CI: the workflow only triggered on tinyml-modelmaker/**
paths and had no step to run torchmodelopt's own tests. A change reintroducing
this bug could merge without any red check.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
self.qconfig_type = qconfig_type is a reference assignment, so the wrapper
module's dict and the caller's dict were the same object. The subsequent
.pop('calibration_dataloader')/.pop('eval_dataloader') therefore mutated
whatever dict the caller passed in, not just the wrapper's own copy -- a
caller that constructs qconfig_type once and reuses or reads it after
construction would see it silently emptied by a constructor call it
doesn't own. Copy before popping.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@musicalplatypus

Copy link
Copy Markdown
Contributor Author

CI failure investigated — pre-existing on upstream/main, unrelated to this PR.

Verified directly against upstream/main's own most recent CI run (same base commit, 331388a): https://github.com/TexasInstruments/tinyml-tensorlab/actions/runs/30376872256 — it already fails identically on macOS + Ubuntu with:

FAILED test_config_validation.py::...test_task_type_is_valid[google_speech_command/config_MSPM0.yaml] - unknown task_type='audio_classification'
FAILED test_config_validation.py::...test_model_name_exists_in_registry[...] - model 'DSCNN_NPU' not in registry

This PR's CI run shows only that same subset — no new failures introduced. It's fixed in #19; should resolve automatically once that merges. (Windows failures don't block merging — the workflow marks that runner continue-on-error: true.)

…ng-fix

# Conflicts:
#	.github/workflows/test-modelmaker.yml
@musicalplatypus
musicalplatypus force-pushed the pr/dataloader-pinning-fix branch from 426f3a0 to 21ffd1d Compare August 6, 2026 04:37
@Adithya-Thonse
Adithya-Thonse merged commit df9422b into TexasInstruments:main Aug 6, 2026
0 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants